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(54) Method and system for processing scan-data from a confocal mircoscope 



(57) The invention discloses a method for process- 
ing scan signals from a confocal microscope and a con- 
focal microscope with a system for processing scan-sig- 
nals. The confocal microscope comprises an illumina- 
tion source (2) and a scanning device (6) with a scan- 
ning mirror system (7). A control and processing unit 
(1 6) is provided, which uses a plurality of programmable 
devices for processing the digital signals in real-time. 
The control and processing unit (16) has at least three 
input ports (16 1t 16 2 , 16 3 ) and one output port (16 4 ). A 
first detector (12) generates analog signals from light 



reflected from a specimen (10) within the microscope 
and a second detector (14) generates analog signals 
proportional to the intensity of the light from the illumi- 
nation source (2). In addition a position signal of the 
scanning laser beam is provided to the control and 
processing unit (16). Analog digital converters (30 1 , 30 2 , 
30 3 ) are located prior to the input ports (16 1( 16 2 , 16 3 ) 
of the control and processing unit (1 6) for providing dig- 
ital signals, generated from the analog signals from the 
first and second detector and the position signal of the 
scanning device. 



CM 
< 

CM 
O 




1~ 

10 



1L 



CL 
UJ 



Printed by Jouve, 75001 PARIS (FR) 



1 



EP1 115 021 A2 



2 



Description 

[0001] This invention relates to a method and a sys- 
tem for processing scandata form a confocal micro- 
scope. More specifically the invention relates to a meth- 5 
od an system which enables real-time processing of the 
scanned data. 

[0002] In confocal microscopy a specimen is scanned 
with a focussed laser beam. The focus of the light beam 
is moved in a section plane of a specimen by tilting two 
scan mirrors around their axes. The axes of the scan 
mirrors are perpendicular to each other. For example 
one scan mirror diverts the laser light in the x-direction 
and the other scan mirror diverts the laser light in the y- 
direction. The power of the reflected or the fluorescent 
light is measured for each scanning point. Each meas- 
ured value, with respect to power, relates to a x, y and 
z-position of the specimen. The user achieves a three- 
dimensional image of the specimen. 
[0003] An example for a confocal microscope is dis- 
eased in the U.S. Patent 5,804,813 to Wang Jyh Pyng 
et al., entitled "Differential confocal microscopy". The 
present invention uses a He-Ne laser as the light source, 
and the focusing device is a microscope objective lens. 
The reflected light from the almost completely reflected 
to an optical detector , which can use photodiodes, av- 
alanche photodiodes, photo multipliers, charge coupled 
devices (CCDs), or fluorescent screens. The signal is 
detected by the optical detector and then amplified by a 
signal amplifier. The amplified signal is recorded by an 
analog-to-digital converter and then stored by a compu- 
ter. The computer generates the three-dimensional im- 
age by using the intensity of the signal combined with 
the respective coordinate of the specimen. Before the 
measurement it is necessary to use the same sample 
to calibrate the relationship between the variation of sig- 
nal intensity and the height of the sample. 
[0004] The ideal scan pattern of the light beam on the 
surface of a specimen is a meander (to scan one line in 
the x-direction with a constant y-position, then stop the 
scan in the x-direction and move the y-position to the 
next scan line; then this line is scanned in the opposite 
x-direction and so on). In the real world the meander 
form does not exist for high scan rates. This is due to 
the inertia of the galvanometric devices and the scan- 
ning mirrors. The real path of the scanning beam has, 
at scan rates (> 100Hz), a sinusodial form which re- 
quires a correction to the ideal path. Several error types 
may occur, for example, a higher velocity of the scan- 
ning point at the turning point of the real path and differ- 
ing shape of the scan path for different scan directions. 
In addition to that, the different run and process times 
for the intensity and the position signal respectively have 
to be considered. 

[0005] As disclosed by the prior art above registration 
and processing of the position signal is normally done 
with an analog circuit, with computers or digital signal 
processors (DSP). There are certain disadvantages of 



signal processing with analog circuits. For example, sig- 
nal processing can only be carried out with a correction 
function implemented in the analog circuit. Changes of 
the correction mechanism, for example change of the 
scan rate, requires a lot of effort. Moreover the accuracy 
of analog circuits with respect to mathematical opera- 
tions reaches a limit at high scan rates. Pixel rates great- 
er than one MHz at a 12 bit accuracy can be achieved 
only with enormous effort. 

[0006] Signal processing with a computer is or DSP 
is possible at low scan rates. Furthermore the signal 
processing is also flexible since the algorithms for 
processing can be changed without a major effort. At 
high scan rates computers or DSP fail due to the lack of 
real-time ability. 

[0007] It is an object of the present invention to pro- 
vide a system for processing scan-signals from a con- 
focal microscope in digital form and real-time. Moreover 
the accuracy of the measurement should be governed 
solely by the accuracy detectors the accuracy of the dig- 
ital analog converters. Furthermore, the processing of 
the scan-signals should be possible at high scanning 
rates. 

[0008] This object is achieved by a system which 
comprises: 

at least three different analog signal are generated the 
by confocal microscope; 

at least three analog digital converters each receiv- 
ing a different analog signal and produces respec- 
tive digital signals at an early stage; and 
a control and processing unit which uses a plurality 
of programmable devices for processing the digital 
signals in real-time. 

[0009] It is also an object of the present invention to 
provide a confocal microscope with a system for 
processing scan-signals which allows the processing of 
the scanned signals at high scanning rates and real 
time. Additionally, there should be a high flexibility which 
allows for example an easy alteration of the used algo- 
rithms. 

[0010] This object is accomplished by confocal micro- 
scope which comprises: a illumination source, 

a scanning device with a scanning mirror system 
a control and processing unit, which uses a plurality 
of programmable devices for processing the digital 
signals in real-time, having at least three input ports 
and one output port, 

a first detector generates analog signals from light 
reflected from a specimen; 
a second detector generates analog signals propor- 
tional to the intensity of the light from the illumination 
source; 

an electrical connection providing an analog posi- 
tion signal, generated by the scanning device, to the 
control and processing unit; and 
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a first, second and third analog digital converter are 
located prior to the input ports of the control and 
processing unit for providing digital signals, gener- 
ated from the analog signals from the first and sec- 
ond detector and the scanning device. 

[0011] Another object of the present invention is to 
provide a method for processing scan-signals from a 
confocal microscope. The method should enable 
processing of the analog scan-signals at high scanning 
rates. Furthermore the method should have a real time 
ability so that a flexible an loss free data collection is 
possible. 

[0012] This object is achieved by a method which 
comprises the following steps: 

generating at least three different analog signals by 
the confocal microscope wherein a first signal rep- 
resents light reflected from a specimen, a second 
signal represents an illumination reference and a 
third signal represents a position on the specimen; 
converting each analog signal with a separate an- 
alog digital converters into respective digital sig- 
nals; 

providing the digital signals to a control and 
processing unit which uses at least one of program- 
mable devices for processing the digital signals in 
real-time; 

buffering the digital position signals in a line-buffer 
of the control and processing unit; 
correcting the first digital signal for intensity fluctu- 
ations of the second digital signal for generating a 
corrected digital detection signal; and 
combining a specific digital position signal with a 
specific digital detection signal. 

[001 3] An advantage of the inventive method and the 
system is that the position signal of the focused scan- 
ning beam and signals from the detectors (scanning 
beam and reference beam) are digitized at a very early 
stage. Processing of the data is done mainly in digital 
form with the use of a programmable control and 
processing unit which is realized by a programmable 
digital circuit (for example FPGA, Field Programmable 
Gate Array). Correction parameters can be used online, 
which would make downstream image processing much 
easier or obsolete. The accuracy of the device is gov- 
erned solely by the accuracy of the detectors and the 
analog digital converters. Analog digital converters with 
a big processing band width are available at reasonable 
costs. The position correction function can be pro- 
grammed with respect to the requirements resulting 
from the measurement conditions. An alteration of the 
position correction function can be done online. The dy- 
namic pixel accumulation records several measurement 
values during the scan of an image pixel and accumu- 
lates the data in a suitable buffer. The dynamic pixel ac- 
cumulation allows a wide dynamic range form a few Hz 



4 

to some MHz (right now the upper limit is 250 MHz) this 
make high scanning rates possible. 
[0014] Pixel accumulation calls for a simple analog 
circuit without giving up the high resolution and the wide 
dynamic range. Despite the fact, that the analog digital 
converters enable only sampling of intensity signals 
from a narrow band of signal heights, the scanning sev- 
eral times and the pixel accumulation result in a theo- 
retical unlimited bit resolution. 
[0015] With the pixel accumulation the electronic 
processing is adjusted in time to the scan -velocity. 
Thereby, during a slow scan, the registered intensity sig- 
nals are accumulated until the scanning beam has 
reached the next pixel element of the object. The pixel 
rate can be adjusted to the internal processing rate by 
pixel accumulation. 

[001 6] The advantages of pixel accumulation are an- 
alog to line accumulation (multiple sequential scan of 
one scan line and addition of the intensity data in a suit- 
able buffer arrangement). Moreover, differences in the 
line-scan in one direction from the line scan in the other 
direction are averaged by line accumulation. In case of 
resonant galcanometers for the x-scan, a pixel accumu- 
lation is not possible so that line accumulation has to be 
used only. 

[0017] A further advantage of the system is its real 
time ability, which repeats predetermined process steps 
with an accuracy up to nanoseconds. The inventive sys- 
tem fulfills the requirements of flexibility and real time 
ability. There is no need for buffer memories, since a 
computer controlled data recording is not used. The real 
time ability of the inventive system guaranties a loss free 
and flexible data collection with the highest, at present 
possible, scan rates. 

[0018] Computers can also be programmed, but they 
are controlled by interrupts so that a running process 
have to be ended before further data processing can go 
on. This is, with respect to the present invention, also 
not considered a real-time ability. The present invention 
has the ability of a high flexibility with respect to data 
process changes and real-time processing of data. 
[0019] The subject matter of the invention is de- 
scribed with reference to the embodiments shown in the 
drawings. 

Fig. 1;shows a schematic illustration of a confocal 
microscope and a system for achieving three di- 
mensional specimen images; 
Fig. 2 is an embodiment of a process unit; 
Fig. 3 shows an ideal path of the focus of the scan- 
ning beam; and 

Fig. 4 shows an actual path of the focus of the scan- 
ning beam. 

[0020] Fig. 1 shows a schematic illustration of a con- 
focal microscope and a system for achieving three di- 
mensional specimen images. An illumination system 2 
generates a light beam L. A beam splitter 4 divides the 
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incident light beam L into afirst path L 1 and into a second 
path L 2 . With the first path L 1 the light from the illumina- 
tion system is directed to a scanning device 6. The scan- 
ning device 6 comprises a scanning mirror system 7 
which is moveable such, that the light of the first path L 1 
is scanned across a specimen 1 0. Before the light of the 
first path L 1 reaches the specimen 10 the light passes 
an optical system 8. The light reflected from the speci- 
men travels on 1 2 the first path L t back to the beam split- 
ter 4. After the beam splitter 4 a first detector 12 is po- 
sitioned, which receives the light reflected from the 
specimen 1 0. The first detector 1 2 converts the received 
light into an first electrical signal I which is proportional 
to the intensity of the received light received form the 
specimen. A second detector 1 4 receives directly via the 
second path L 2 the light from the illumination system 2. 
As the first detector 1 2 the second detector 14 converts 
the received light into an second electrical signal R 
which is proportional to the intensity of the received light, 
which is a illumination reference. A control and process- 
ing unit 1 6 is provided which has a first second and third 
input port 1 e 1 , 16 2 and 16 3 . The first electrical signal I 
is fed to the first input port 1 6 1 and the second electrical 
signal R is fed to the second input port 16 2 . The third 
input port 1 6 3 receives via an electrical connection 1 8 a 
position signal P which is generated in the scanning de- 
vice 6. The disclosed embodiment describes three dif- 
ferent analog signals I, R and P which are fed to the 
control and processing unit 16. With the use of multi- 
plexers (not shown) more than three different signal can 
be provided to and processed with the control and 
processing unit 16. 

[0021] The control and processing unit 16 converts 
the incoming analog signals (first electrical signal I, sec- 
ond electrical signal R and position signal P) which may 
be distorted and disrupted in into corrected digital sig- 
nals. Via an exit port the digital signals are sent to a com- 
puter 20 which may carry out some image processing. 
A conventional display 22 is used to show the image of 
the specimen 1 0 to an user of the system. 
[0022] An embodiment of the control and processing 
unit 16 is shown in Fig. 2. The control and processing 
unit 16 is realized with a plurality of FPGA-units (Field 
Programmable Gate Array). The first analog electrical 
signal I enters a first analog-digital converter 30 1( the 
second analog electrical signal R enters a second ana- 
log-digital converter 30 2 and the analog position signal 
P enters a third analog-digital converter 30 3 . From the 
first analog-digital converter 30<| and from the second 
analog-digital converter 30 2 the digitized signal are fed 
to a first and a second delay-modules 32 and 34 respec- 
tively which are used to carry out run-time differences. 
The use of the first and second delay-module 32 an 34 
is important since the first and second electric signal I 
and R have a processing time which is different to the 
position signal P. This means that the reaction time of 
thescanning mirror system 7 is delayed due to its inertia. 
The delag modules eliminate the possible error due to 



its own processing time and reaction time differences. 
The output of the first and second delay module 32 and 
34 is fed to the first and second input port 16^ and 16 2 
respectively of the control and processing unit 1 6. Here 

5 the digital signal from the first delay-module 32 is sent 
to a first adder 40 followed by a first buffer 42. There is 
a loop 70 between the first buffer42 and the first adder 
40 to enable a pixel accumulation. From the first buffer 
42 the data are sent to a first look-up-table 43 which 

10 carries out a log-conversion of the data. From the sec- 
ond delay-module 34 the digital signal is sent to a sec- 
ond adder 50 followed by a second buffer 52. There is 
a loop 80 between second the buffer 52 and the second 
adder 50 to enable a pixel accumulation. From the sec- 

15 ond buffer 52 the data are sent to a second look-up-table 
53 which carries out a log-conversion of the data. From 
the third analog-digital converter 303 the positional data 
are sent to a third adder 60 followed by a third buffer 62. 
There is a loop 90 between the third buffer 62 and the 

20 third adder 60 to enable a pixel accumulation. From the 
third buffer 62 the data are sent to an address look-up- 
table 63 which carries out correction of the position data 
with respect to the scan along the x-direction. In order 
to obtain the correct digital data for the position of the 

25 scan beam there may be different look-up tables, one 
for each direction of the scan in the x-direction. 
[0023] From the first look-up-table 43 and the second 
look-up-table 53 the data are fed to a calculation device 
44. The calculation device 44 subtracts the data repre- 

30 sentative of the first electrical signal I from the data rep- 
resentative of the second electrical signal R. This ena- 
bles a correction of the short term intensity fluctuations 
of the laser output signal. The data from the calculation 
device 44 are fed to a fourth look-up-table 45, which car- 

35 ries out a conversion of the data with respect to the basis 
e. The result of this conversion is that the data are avail- 
able as the quotient of data representing the first elec- 
trical signal I and data representing the second electrical 
signal R. The data from the forth look-up table 45 are 

40 fed to a forth adder and from there to a line buffer 47 
which correlates a specific digital position signal with a 
specific digital detection signal. There is also a loop 48 
between the line buffer 47 and the forth adder 46 to en- 
able a line accumulation. The position data are from the 

45 address look-up-table 63 are sent also to the line buffer 
47. From the line buffer 47 the corrected digital signals 
are sent to via the exit port 164 of the processing unit 
16 to a data input port 100 of the computer 20. 
[0024] In most cases the devices used to tilt the scan- 

so ning mirror system 7 are galvanometers. In general, 
both resonant and non-resonant galvanometers are 
used. The resonant galvanometers are very fast (ap- 
prox. several kHz), but they are not as exact as the non- 
resonant galvanometers, which are much slower. Fig. 3 

55 shows and ideal path 110 of the focus of the scanning 
beam across specimen area of interest 112. The dashed 
line represents the ideal path 110. The ideal path 110 
comprises a plurality of linear movements 1 1 4 of the fo- 
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cused beam in a direction x of the specimen area 112. 
The linear movements 1 1 4 are separated by a constant 
line feed 116 which has a direction y, perpendicular to 
the linear movements 1 1 4. In other words the ideal path 
1 1 0 of the focus of the scanning beam would be a back 
and forth pattern of parallel lines in the direction x which 
are separated by the constant line feed 1 1 6 in the direc- 
tion y. The line feed 116 is done by tilting the y-mirror 
(not shown), thereby moving the focus of the scanning 
beam to the next line. This next line is scanned with the 
x-mirror (not shown) followed by another line feed 116 
and so on until the whole specimen area 1 1 2 is scanned. 
[0025] The ideal path 1 1 0 as shown in Fig. 3 can not 
be achieved, because of the inertia of the mirrors and 
the and the moving galvanometer elements. Actually, a 
real path 120 across the specimen area of interest 112 
is shown in Fig. 4. The form of the real path is sinusoidal. 
According to the differences between the ideal path 1 1 0 
and the actual path 1 20 a correction of this difference is 
necessary. Furthermore, it has to be considered that the 
velocity of the focus of the scanning beam in the direc- 
tion x increases at turning points 122 compared to linear 
regions 124 of the sinusoidal curve. In order to run a 
microscope at considerable high scanning rates, a real 
time processing unit (field programmable gate array, 
FPGA) is used to correct and process the detector and 
position signals (for example, to provide a correction for 
the differences as shown in Fig. 3 and Fig. 4). 
[0026] The FPGA provides also a correction in the 
case of an asymmetric scan path (the scan path in the 
x-direction differs from the scan path in the opposite di- 
rection). This requires a different correction for the scan 
path in the x-direction and another correction for the 
scan path in the opposite direction. In other words, the 
position signal has to be corrected with respect to the 
different paths of the scanning beam across the object. 
[0027] The invention has been described in detail with 
particular reference to certain preferred embodiments 
thereof, but it will be understood that variations and 
modifications can be effected within the spirit and scope 
of the invention. 
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1 . A system for processing scan-signals from a confo- 
cal microscope, the system characterized in that: 

at least three different analog signal (I, R, P) 
are generated the by confocal microscope; 
at least three analog digital converters (30^ 
30 2 , 30 3 ) are provided for receiving a different 
analog signal and producing respective digital 
signals at an early stage; and 
a control and processing unit (1 6) uses at least 
one programmable device for processing the 
digital signals in real-time. 
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2. The system as recited in claim 1 wherein the pro- 
grammable device is a field programmable gate ar- 
ray. 

3. The system as recited in claim 1 wherein the three 5 
different analog signals (I, R, P) are generated by 
the confocal microscope, the three different signals 
comprise a first electrical signal (30^ which is pro- 
portional to the intensity of the light received form 
the a specimen (1 0), a second electrical signal (30 2 ) 
which is an illumination reference and a third elec- 
trical signal (30 3 ) which is a position signal of an 
actual position of a scanning light beam on the 
specimen (10). 

4. The system as recited in claim 3 wherein the control 
and processing unit (16) comprises a address look 
up table (63) to provide a correction for the position 
signal (P). 

5. The system as recited in claim 4 wherein the posi- 
tion signal (P) generated from a scan in the x-direc- 
tion is different from a scan in the opposite direction 
x-direction with respect to time, and the correction 
of the position signal (P) is chosen with respect to 
the scan direction. 

6. The system as recited in claim 1 wherein the control 
and processing unit (16) comprises a calculation 
device (44) to correct short term intensity fluctua- 
tions, and a line buffer (47) to correlate a specific 
digital position signal with a specific digital detection 
signal. 

7. The system as recited in claims 1 to 6 is incorporat- 
ed in a confocal microscope, wherein the confocal 
microscope comprising: 

an illumination source (2); 
a scanning device (6) with a scanning mirror 
system (7); 

a first detector (12) generates analog signals 
(I) from light reflected from the specimen (10); 
a second detector (14) generates analog sig- 
nals (R) proportional to the intensity of the light 
from the illumination source (2); and 
an electrical connection (18) providing an ana- 
log position signal (P) generated by the scan- 
ning device, to the control and processing unit 
(16). 

8. A method for processing scan-signals from a con- 
focal microscope, is characterized by the following 
steps: 

generating at least three different analog sig- 
nals (I , P, R) by the confocal microscope where- 
in a first signal (I) represents light reflected from 



a specimen (10), a second signal (P) repre- 
sents an illumination reference and a third sig- 
nal (R) represents a position on the specimen 
(10); 

converting each analog signal with a separate 
analog digital converter (30^ 30 2 , 30 3 ) into re- 
spective digital signals; 
providing the digital signals to a control and 
processing unit (16) which uses a plurality of 
programmable devices for processing the dig- 
ital signals in real-time; 

buffering the digital position signals in a line- 
buffer of the control and processing unit (16); 
correcting the first digital signal (I) for intensity 
fluctuations of the second digital signal (R) for 
generating a corrected digital detection signal; 
and 

combining a specific digital position signal with 
a specific digital detection signal. 

The method as recited in claim 8 wherein the control 
and processing unit (16) comprises programmable 
devices which are field programmable gate arrays. 

The method as recited in claim 8 comprises an ad- 
ditional step of providing a correction forthe position 
signal (P) with the control and processing unit (16), 
wherein the correction is implemented in a address 
look up table (63). 

The method as recited in claim 10 wherein the cor- 
rection of the position signal (P) is done by providing 
a correction forthe scan in the x-direction and a dif- 
ferent correction for the scan in the opposite x-di- 
rection. 

The method as recited in claim 8 wherein a first 
look-up table (43) carries out a log conversion of the 
first electrical signal (I). 

The method as recited in claim 8 wherein a second 
look-up table (53) carries out a log conversion of the 
second electrical signal (R). 

The method as recited in claim 8 comprises the 
steps of: providing the combined digital position sig- 
nal and the digital detection signal to a computer 
(20); and 

displaying and image of the scanned speci- 
men area. 
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